Using GCP Media CDN with private AWS storage buckets
Context
Google’s Media CDN is Google Cloud’s Media delivery CDN platform designed for streaming video and large object downloads . With this platform ; our customers get access to massive coverage and presence globally in over 206+ countries/territories and 1300+ cities. For more details about Media CDN , please refer this link
While Media CDN is a delivery platform ; the origin can be anywhere / on any platform . Example ; the origin can be Google Storage , it can be AWS S3 etc . The origin buckets can be public buckets or private buckets . Customers tend to make buckets private in order to implement use-cases like blocking large public request volumes hitting the bucket or restricting access to media assets to authorized users.
This blog focuses on how Media CDN supports AWS Signature Version 4 tp connect to private Amazon Simple Storage Service (S3) buckets.
What is covered / not covered in this blog
Covered
- AWS private bucket configurations
- GCP Media CDN origin configurations
- GCP Media CDN service configuration
Not covered
- This blog does not cover explanation into GCP Media CDN constructs like service , origin etc.
- AWS public bucket configuration
- Detailed explanation of Media CDN services configuration
- Other private buckets (like Google cloud private bucket)
- Certificate setup to make client https calls
I have broken down this blog in following main sections -
Section 1 will cover AWS side configurations.
Section 2 will cover GCP side configurations.
Section 3 will cover basic verification details.
Let’s start with real configuration level stuff !!
AWS side configuration steps
STEP 1
Create a Private bucket (example below)
This step highlights the creation of a private bucket in AWS , which will hold the content to be served by GCP Media CDN . This is pretty straight forward .. just ensure that bucket level setting mentioning block all public access is in place.
Ensure that the objects in bucket are not all public
STEP 2
Go to Identity Management → Users tab → Add user
- Don’t add this user to any group
- Please go ahead (move next and finish)
- You should see a screen which shows following 2 keys
(a) Access Key ID
(b) Secret access key (note / save this as we need this at a later stage)
STEP 3
Next , we want the user created in previous step to have the access to bucket with only permissions as ‘ s3.GetObject’ . In order to get this step done , please go back to AWS private S3 bucket and edit the bucket policy .
Below is the sample policy that I wrote
{
“Version”: “2012–10–17”,
“Id”: “Policy1663216626024”,
“Statement”: [
{
“Sid”: “Stmt1663216609692”,
“Effect”: “Allow”,
“Principal”: {
“AWS”: “arn:aws:iam::100552525497:user/gaurav.google”
},
“Action”: “s3:GetObject”,
“Resource”: “arn:aws:s3:::gaurav-google-s3-private/*”
}
]
}
With this we are done with AWS side configuration and we are all set to configure the GCP Media CDN level configurations
GCP Side configuration steps
STEP 4
Secret Manager configuration
We need to upload the AWS side secret access key (step 2 above) in GCP secret manager
- Go to GCP secret manager — create secret — give it any name
2. Once secret is created — create a version and upload / paste the AWS secret access key
You will see your version created ( Please dont get confused by below snip .. I have 2 versions as an example . In this blog , I am using version 2 )
STEP 5
This step focuses on creating the GCP Media CDN origin . I am doing this step using Google cloud shell . For this ; I will create a YAML file with required parameters and import this YAML file to create the Origin .
Sample origin file YAML format is shown below
name: << put name>
originAddress: “<< your aws bucket name >>”
retryConditions: [HTTP_5XX, NOT_FOUND]
protocol: HTTPS
awsV4Authentication:
accessKeyId: “you get this from aws console “
secretAccessKeyVersion: “projects/GCPPROJECT/secrets/SECRET/versions/VERSION”
originRegion: “<<aws region>>”
An example of YAML file is as follows →
name: private-s3-origin
originAddress: “gaurav-google-s3-private.s3.ap-south-1.amazonaws.com”
retryConditions: [HTTP_5XX, NOT_FOUND]
protocol: HTTPS
awsV4Authentication:
accessKeyId: “ABCDEFGHIJKLMNOP”
secretAccessKeyVersion: “projects/7142121/secrets/gauravmadan_gcs_access_key/versions/2”
originRegion: “ap-south-1”
STEP 6
- Create MEDIA CDN Origin
- We need to now create media CDN origin by using YAML file created in step 5
- Execute the following command on GCP command
# gcloud edge-cache origins import private-s3-origin — source=name-of-yamlfile.yaml
Verify in the Google cloud console UI that origin is created as follows
STEP 7
Once the origin is created , the next step is to create the MEDIA CDN service . This service will call the origin configured in previous step .
Although this blog will not cover details of each and every configuration parameters required to configure media service on GCP platform ; however I will share the sample configuration and steps to configure the same . You will find a detailed blog on this soon !
Sample YAML file for MEDIA CDN service configuration (aws-service.yaml)
name: gm-aws-service
routing:
hostRules:
- hosts:
- ‘*’
pathMatcher: routes
pathMatchers:
- name: routes
routeRules:
- priority: 1
matchRules:
- prefixMatch: /
origin: private-s3-origin
routeAction:
urlRewrite:
hostRewrite: gaurav-google-s3-private.s3.ap-south-1.amazonaws.com
pathPrefixRewrite: “/”
cdnPolicy:
cacheMode: CACHE_ALL_STATIC
defaultTtl: 3600s
headerAction:
responseHeadersToAdd:
- headerName: “x-cache-status”
headerValue: “{cdn_cache_status}”
Call this YAML file to configure MEDIA CDN service using following Google cloud command
# gcloud edge-cache services import aws-service — source=aws-service.yaml
Media CDN service can be configured using GCP console UI as well , but I took an example of doing this via a YAML file . Readers may take an opportunity to do the same stuff using GUI as well.
STEP 8
Since MEDIA CDN service will be reading from GCP ‘Secret Manager’ , please ensure that the service account for MEDIA CDN has the IAM role of ‘Secret Manager Secret Accessor’
Verification steps
Here are a few commands that you can use to verify the configuration is fine -
# gcloud edge-cache origins describe <name-of-your-origin>
# gcloud edge-cache services describe <name-of-service>
From functionality point of view ; browse to object in private S3 bucket using following format
http:// <media cdn IP> / <object-in-s3 >
- On successful configuration , You will get the MEDIA CDN IP from output of command gcloud edge-cache services describe <name-of-service> OR you can get the same from GCP console UI .
- Feel free to make changes in your DNS if you want to access using a domain name instead of IP address of MEDIA CDN
- I am using http here because in this blog I had not setup https access , but that doesn’t change media CDN functionality in any shape of form
If you have carefully noticed in this example of service YAML file ; I have added a custom response header as follows -
- headerName: “x-cache-status”
headerValue: “{cdn_cache_status}”
So users should be able to see this header added in the response served . You may see something like below (based on your location and content )
That's it for now ! Hope this was useful and you are able to / will be able to configure GCP MEDIA CDN to fetch contents from private AWS S3 buckets
Useful References
https://cloud.google.com/media-cdn/docs/connect-to-s3-compatible-buckets
Disclaimer: This is to inform readers that the views, thoughts, and opinions expressed in the text belong solely to the author, and not necessarily to the author’s employer, organization, committee or other group or individual.
